Acquisition strategy for neural network based image restoration

ABSTRACT

Methods and systems for neural network based image restoration are disclosed herein. An example method at least includes acquiring a plurality of training image pairs of a sample, where each training image of each of the plurality of training image pairs are images of a same location of a sample, and where each image of the plurality of training image pairs are acquired using same acquisition parameters, updating an artificial neural network based on the plurality of training image pairs, and denoising a plurality of sample images using the updated artificial neural network, where the plurality of sample images are acquired using the same acquisition parameters as used to acquire the plurality of training image pairs.

FIELD OF THE INVENTION

The invention relates generally to artificial intelligence (AI) enabledimage restoration, and specifically to AI enabled noise reduction andsparse reconstruction.

BACKGROUND OF THE INVENTION

In many types of microscopy, noise is difficult to remove or reduce,which leads to less than desirable results. Further, the types of imagesusers desire may make the noise even more difficult to manage. Forexample, in charged particle microscopy, the parameters of the chargedparticle beam and other image acquisition parameters can affect thenoise in an image, but also may have adverse effects on the desiredfield of view and sample interaction. To further the example, a tradeoffbetween charged particle beam dwell time and noise exists. Specifically,short dwell times may be preferred to ensure there are no or minimalchanges during image acquisition, such as drift. Short dwell times mayalso lead to quicker image acquisitions. A downside, however, is thatimages from short dwell times tend to be noisy and/or sparse, e.g.,having pixels void of information, which is typically undesirable by enduser. While long dwell times may reduce noise, e.g., increase signal tonoise, and provide a less sparse image, the longer dwell times tend toincrease image acquisition times and may further result in sampledamage, especially with biological samples. Long dwell times typicallyincur drift during image acquisition, as well. These tradeoffs andissues are just a few that may affect noisy images.

End users typically desire large fields of view and/or three-dimensionalvolumes, which require long acquisition times (on the scale of a monthin some examples) that is only further increased if long dwell times areused. As such, fast acquisition times with reduced noise are desirable.The acquisition time could be further reduced with sparse imageacquisition. While various attempts for image restoration have been madeover the years to solve this problem, the offered solutions have theirown drawbacks, such as images overly smoothed or virtually unimprovedimages. As such, there remains a desire to solve the noisy imageproblem.

SUMMARY

Apparatuses and methods for neural network based image restoration aredisclosed herein. Image restoration includes denoising images, sparsereconstruction of images, and combinations thereof. An example methodfor neural network based denoising at least includes acquiring aplurality of training image pairs of a sample, where each training imageof each of the plurality of training image pairs are images of a samelocation of a sample, and where each image of the plurality of trainingimage pairs are acquired using same acquisition parameters, updating anartificial neural network based on the plurality of training imagepairs, and denoising a plurality of sample images using the updatedartificial neural network, where the plurality of sample images areacquired using the same acquisition parameters as used to acquire theplurality of training image pairs.

An example apparatus for implementing neural network based denoising atleast includes a charged particle microscope imaging platform; and acontroller at least coupled to control the charged particle microscopeimaging platform. The controller is coupled to a non-transitory computerreadable medium including code that, when executed by the controller,causes the system to acquire a plurality of training image pairs of asample, where each training image of each of the plurality of trainingimage pairs are images of a same location of a sample, and where eachimage of the plurality of training image pairs are acquired using sameacquisition parameters, update an artificial neural network based on theplurality of training image pairs, the artificial neural networkincluded with or coupled to the system, and denoise a plurality ofsample images using the updated artificial neural network, where theplurality of sample images are acquired using the same acquisitionparameters as used to acquire the plurality of training image pairs.

Another embodiment of neural based image restoration is directed towardsparse reconstruction. An example method of sparse reconstruction atleast includes acquiring a plurality of training image pairs of asample, where each training image of each of the plurality of trainingimage pairs are images of a same location of a sample but includedifferent image pixels, and where each image of the plurality oftraining image pairs are acquired using same acquisition parameters,updating an artificial neural network based on the plurality of trainingimage pairs, and reconstructing each of a plurality of sample imagesusing the updated artificial neural network, where the plurality ofsample images are acquired using the same acquisition parameters as usedto acquire the plurality of training image pairs, and where each imageof the plurality of sample images are sparse images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a charged particle microscope system inaccordance with an embodiment of the present disclosure.

FIG. 2 is an example method for denoising images in accordance with anembodiment of the present disclosure.

FIG. 3 is an example method for sparse image reconstruction inaccordance with an embodiment of the present disclosure.

FIG. 4 is an example image sequence showing denoising of images inaccordance with an embodiment of the present disclosure.

FIG. 5 is an example image sequence showing sparse reconstruction inaccordance with an embodiment of the present disclosure.

FIG. 6 is an example functional block diagram of a computing system 600in accordance with an embodiment of the present disclosure.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention relate to neural network basedimage restoration. In one example, an artificial neural network may betrained on a small number of noisy images to be a denoiser, which isthen used to denoise a larger data set of sample images. In such anexample, the acquisition parameters used to acquire the small number ofnoisy images is also used to acquire the sample images, which ensuresthat the noise in both image sets is similar. In another example, anartificial neural network may be similarly trained to provide sparsereconstruction instead of denoising. In the sparse reconstructionexample, the training images include sparse images that at leastslightly overlap so that the network learns how to fill in the sparseimages of a large data set of sparse sample images. However, it shouldbe understood that the methods described herein are generally applicableto a wide range of different AI enhanced or enabled image restorationtechniques, and should not be considered limiting.

As used in this application and in the claims, the singular forms “a,”“an,” and “the” include the plural forms unless the context clearlydictates otherwise. Additionally, the term “includes” means “comprises.”Further, the term “coupled” does not exclude the presence ofintermediate elements between the coupled items. Additionally, in thefollowing discussion and in the claims, the terms “including” and“comprising” are used in an open-ended fashion, and thus should beinterpreted to mean “including, but not limited to . . . .” The term“training image pairs” or “training pairs” refers to pairs of images ofa same location of a sample and acquired using the same acquisitionparameters. As used herein, the term “acquisition parameters” refers tosettings of a charged particle microscope used to acquire one or moreimages, and at least includes beam dwell time per pixel, beam spot size,beam landing energy, but does not include the number of images acquiredor a grid size used in acquiring the images. In general, the“acquisition parameters” of discussion herein are mainly thoseparameters that control the magnitude, including time, of theinteraction of the charged particle beam with the sample. Additionally,the term “sparse training image pairs” or “sparse training pairs” refersto pairs of images of the same location on the sample, but which containdifferent patterns of scanned pixels, which are obtained using the sameacquisition parameters.

The systems, apparatus, and methods described herein should not beconstrued as limiting in any way. Instead, the present disclosure isdirected toward all novel and non-obvious features and aspects of thevarious disclosed embodiments, alone and in various combinations andsub-combinations with one another. The disclosed systems, methods, andapparatus are not limited to any specific aspect or feature orcombinations thereof, nor do the disclosed systems, methods, andapparatus require that any one or more specific advantages be present orproblems be solved. Any theories of operation are to facilitateexplanation, but the disclosed systems, methods, and apparatus are notlimited to such theories of operation.

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed systems, methods, and apparatus can be used in conjunctionwith other systems, methods, and apparatus. Additionally, thedescription sometimes uses terms like “produce” and “provide” todescribe the disclosed methods. These terms are high-level abstractionsof the actual operations that are performed. The actual operations thatcorrespond to these terms will vary depending on the particularimplementation and are readily discernible by one of ordinary skill inthe art.

In general, charged particle images, for example, include variousamounts of noise. The various amounts of noise may be mostly based on adwell time of the probing charged particle beam, but other factors mayalso influence the noise as well. In terms of dwell time, the longer thedwell time the less noise included, e.g., the greater the signal tonoise ratio (SNR), but the shorter the dwell time the greater the noise,e.g., lower the SNR. While this tradeoff seems straightforward toacquire less noisy images, the type of sample and the volume of imagesdesired additionally affects the dwell time calculus. For example, asoft, delicate sample, e.g., a biological sample, may be damaged by longdwell times. It should also be noted that biological samples maytypically suffer worse SNR due to the lack of contrast in the image,which may counsel for longer dwell times. Further, the number of imagesneeded to image a large area or a volume of a sample may be greatlyincreased by long dwell times. In some instances, a large number ofimages of a soft sample may be desired, which may result in long dwelltimes to account for the additional noise of the soft sample. Such animaging workflow may lead to a month of image acquisition. Clearly, thisis undesirable and a quicker, less damage prone image acquisition isdesirable.

While prior solutions to the noise problem are available, the resultsare typically less than desirable. Such solutions may include classicaldenoising algorithms and even more recent deep learning based solutions.The classical approach typically is based on oversimplified assumptionsabout the actual noise in the sample images and result in over smoothingof the image. As such, information is lost. As for the deep learningapproaches, the training data is typically not specific enough duemainly to training on different noise, which may be artificiallydistorted clean images. Additionally, these deep learning baseddenoising networks are trained with noisy images and associate cleanimages, but the associated noisy and clean images are acquired underdifferent settings, which affect the network's denoising capabilities.Clearly, newer and better solutions are desired.

One solution to the above disclosed problem includes updating thetraining of a pre-trained blind denoiser, e.g., pretrained blindartificial neural network, using only noisy images (training imagepairs) that include noise similar to that of sample images needingdenoised. The pre-trained blind denoiser may be lightly trained usingnoisy and clean images, for example, but the updating may only use noisyimage pairs that have system specific noise, which helps the denoiserprovide clean images after being updated. Additionally, a large data setof sample images may be acquired under the same acquisition parameter,e.g., charged particle microscope system settings, as used to acquirethe training image pairs. After the sample images are acquired, theupdated denoiser is used to denoise the sample images.

Some advantages of this solution over previous solutions is the use of asmall set of training images, which both improves the denoisingcapability of the denoiser and increases the training time. For example,the update training of the denoiser may take less than an hour. Further,by using short dwell times for both the training image pairs and thelarge data set of sample images, the overall time from image to data maybe greatly increased. Moreover, the denoising may be much better than aconventional deep learning denoiser due to training on images that havesystem specific noise that the denoiser will encounter in the large dataset of sample images. As such, the denoised images will be of betterquality than they would otherwise.

As previously noted, this technique may also be used to reconstructsparse images with only a small change in the training image pairs.Whereas the training image pairs for the denoiser are of a same locationof a sample, the sparse training image pairs may have some overlap andsome non-overlapping areas, which provides training that allows thesparse reconstruction network, e.g., artificial neural network, to fillin the sparse (blank) areas of an image.

FIG. 1 is an example of a charged particle microscope system 100 inaccordance with an embodiment of the present disclosure. The chargedparticle microscope (CPM) system 100, or simply system 100, at leastincludes a CPM environment 102, a network 104, one or more servers 106,and an artificial neural network 114. The CPM system 100 may be used toinvestigate and image samples of various size and makeup. For oneexample, the CPM system 100 may be implemented, at least partially, at acommercial or research site and used to image various samples at thesite. In some embodiments, the CPM system 100 may be distributed acrossvarious locations. For example, the CPM environment 102 may be locatedat a development location, the network 104 distributed locally,regionally, or nationally, and the server 106 located at a server farmand coupled to the CPM environment 100 via the network 104. Regardlessof the organization of the CPM system 100, the system 100 may at leastbe used to implement one or more artificial neural networks (ANN) 114along to perform various image restoration tasks, such as denoisingand/or sparse reconstruction.

The CPM environment 102 includes any type of charged particlemicroscope, but the application of the neural network and imagerestoration techniques disclosed herein is not limited to chargedparticle microscopy, which is used for illustrative purposes only.Example CPMs include scanning electron microscopes (SEMs), transmissionelectron microscopes (TEMs), scanning transmission electron microscopes(STEMs), focused ion beams (FIBs), and dual beam (DB) systems thatinclude both electron and ion beam capabilities, to name a few. The CPMenvironment 102 may be used to obtain electron or ion images of samples,some of which may be images of sequential slices of a sample such as toimage a volume of a sample. Volume imaging may also be referred to sliceand view, which includes imaging a surface of a sample, removing thatsurface, and imaging the newly exposed surface, which is repeated untila desired volume of the sample is imaged. The CPM environment 102 mayinclude various aspects that can be contained in a single tool or thatmay be situated in separate tools. For example, the CPM environment 102may include an imaging platform 108, e.g., an SEM, TEM, or STEM, asample preparation platform 110, and one or more controllers 112. Ofcourse, each platform 108 and 110 may include more than onemicroscope/sample preparation tools as well.

The imaging platform 108 is used to obtain images of samples, some ofthe samples may have been prepared by the sample prep platform 110, butthat is not necessary. The images are obtained using an electron and/orion source to irradiate the sample with a respective beam of chargedparticles. For example, an ion and/or electron beam column provides therespective beam of ions and/or electrons. In some examples, the chargedparticle beam imaging is obtained by a scanned beam, e.g., moved acrossthe sample such as by rastering the beam, while other examples thecharged particle beam is not scanned. Backscattered, secondary, ortransmitted electrons, for example, are then detected and gray scaleimages formed based thereon. The images include gray scale contrastdepending on the materials of the sample, where the changes in grayscale indicate changes in the material type or crystal orientation. Theimaging platform 108 may be controlled by internal controls (not shown),controller 112, or a combination thereof.

The sample prep platform 110 forms some of the samples that are imagedby the imaging platform 108. Of course, imaged samples may also beformed by other tools (not shown). The sample prep 110 may, for example,be a DB system that uses a FIB to prepare and assist in the removal of athin sample from a larger sample, such as by ion milling, ion inducedetching, or a combination thereof, and other processes to process thesample for imaging. Other processes may include, but are not limited to,planarizing mills/etches, fiducial generation, cross-section formation,top-down lamella preparation, etc. The sample prep platform 110 may alsoinclude an electron imaging component that allows the sample prepprocess to be monitored, but the electron imaging component is notrequired. In some embodiments, the sample prep platform 110 may includeother physical preparation aspects—lasers, cutting tools, resinencapsulation tools, cryogenic tools, etc.—that are used to prepare thesample for the imaging platform 108. The sample prep platform 110 may becontrolled by internal controls (not shown), controller 112, or acombination thereof.

The network 104 may be any kind of network for transmitting signalsbetween the CPM environment 102 and the server(s) 106. For example, thenetwork 104 may be a local area network, a large area network, or adistributive network, such as the internet, a telephony backbone, andcombinations thereof.

The servers 106 may include one or more computing platforms, virtualand/or physical, that can run code for various algorithms, neuralnetworks, and analytical suites. While not shown, a user of the CPMenvironment 102 may have access to the servers 106 for retrieval ofdata, updating software code, performing analytical tasks on data, etc.,where the access is through the network 104 from the user's localcomputing environment (not shown). In some embodiments, the useraccesses image data stored on the servers 106, implements imagerestoration, e.g., denoising and/or sparse reconstruction, using the ANN114 (which may be executed on the servers 106 or the CPM Environment102) from their local computing environment.

In operation of a denoising embodiment, an imaging platform 108 is setto acquire images using desired acquisition parameters. The acquisitionparameter may at least determine charged particle beam energy, beam spotsize, and dwell time. Example dwell times may be from 0.3 μs to 3.0 μs.The dwell time determines how long the charged particle beam willimpinge the sample at each pixel of the acquired image, and a shortdwell time will reduce system drift, beam damage, and charging of thesample, to name a few benefits. As used herein, “pixel” may refer to alocation within a viewable area of an acquired image and the beam may bemoved from pixel to pixel to acquire images within the viewable area.The viewable area may be adjustable based on a desired field of view.Once the acquisition parameters are set, a small number of trainingimage pairs of the sample are acquired. Each training image pair willinclude at least two images of a same location on the sample, which mayalso be called spot images or spot training pairs. Acquiring two or morespot training images of the same location provides information about thecharged particle specific system noise. In some embodiments, the numberof training image pairs can be from 100 to 1000, e.g., 200 to 2000 totalimages. The number of training image pairs acquired may be adjustedbased on how close the noise in pre-training the ANN 114 is to thesystem specific noise in the training image pairs. If the noise isclose, then fewer training image pairs may be needed, and vice versa.

After the small number of training image pairs are acquired, thepre-trained ANN 114 may be updated to fine tune the ANN 114 to thesystem specific noise in the training image pairs. The updating of theANN 114 may only use the training image pairs, which due to the shortdwell time will be noisy images. Further, the training image pairs willnot be labeled as is customary for training artificial neural networks.Additionally, no associated clean images will be used in updating thetraining of the ANN 114. As such, ANN 114 may use back propagation foreach training image pair to learn the noise of the system. This trainingtechnique may be referred to as noise-to-noise training, which takesadvantage of the statistical nature of noise. The statistical nature ofnoise is based on the stance that many noise sources have a zero mean.As such, an average of multiple images of a same location may suppressthe noise, revealing a clean image. Moreover, training on the smallnumber of training image pairs allows for a fast training due to fastconvergence that leads to a good match with the system specific noise.

Before, after or in parallel with updating the training of the ANN 114,a large data set of sample images may be acquired using the sameacquisition parameters. This large data set may result in terabytes ofdata (e.g., 10,000 plus images) that need to be denoised. Due to usingthe same fast dwell time, the time to acquire all of the sample imagesmay be much less than conventional large scale data set acquisitions.Post-acquisition of the sample images and updating the ANN 114, thesample images may be denoised. This denoising step may be performed atthe system 100, or at the serer 106, and may be initiated by a user atany time.

While the above example operation was discussed in the denoisingenvironment, the sparse reconstruction technique could similarly beimplemented with changed made to the sparse training image pairs asnoted above. Other than the differences between the training images, theprocess of denoising and sparse image reconstruction (which cancollectively be referred to as image restoration) is similar, if not thesame. In terms of sparse reconstruction, sparse images, e.g., imageswith missing data, may be completed, e.g., the missing data filled in,by a trained artificial neural network.

While the images provided to the ANN 114 is described as being obtainedby imaging platform 108, in other embodiments, the images may beprovided by a different imaging platform and provided to the ANN 114 viathe network 104. For example, both the training image pairs and sampleimages may be provided to the servers 106 via the network, then theupdating of the ANN 114 may be performed using the training image pairsprior to denoising the sample images. As long as the acquisitionparameters used to acquire the training images and the sample images,the charged particle system used for imaging does not need to beco-housed with the ANN 114.

In one or more embodiments, the ANN 114, which may also be referred toas a deep learning system, is a machine-learning computing system. TheANN 114 includes a collection of connected units or nodes, which arecalled artificial neurons. Each connection transmits a signal from oneartificial neuron to another. Artificial neurons may be aggregated intolayers. Different layers may perform different kinds of transformationson their inputs.

One type of ANN 114 is a convolutional neural network (CNN). A CNN isconventionally designed to process data that come in the form ofmultiple arrays, such as a color image composed of three two-dimensionalarrays containing pixel intensities in three color channels. Examplearchitecture of a CNN is structured as a series of stages. The first fewstages may be composed of two types of layers: convolutional layers andpooling layers. A convolutional layer applies a convolution operation tothe input, passing the result to the next layer. The convolutionemulates the response of an individual neuron to visual stimuli. Apooling layer combines the outputs of neuron clusters at one layer intoa single neuron in the next layer. For example, max pooling uses themaximum value from each of a cluster of neurons at the prior layer. Inone or more embodiments, the ANN 114 is a CNN configured to reconstructan image, such as through denoising the image and/or sparsereconstruction of the image.

FIG. 2 is an example method 200 for denoising images in accordance withan embodiment of the present disclosure. The method 200 may beimplemented in part by a charged particle microscope system and furtherin part on one or more processing cores able to receive data from thecharged particle microscope system. An example charged particlemicroscope system being system 100. Of course, other charged particlemicroscope systems could be used as well. In general, the method 200retrains or updates an ANN, such as the ANN 114, based on a small numberof training image pairs, then a large data set of sample images may bedenoised using the retrained/updated ANN.

The method 200 may begin at process block 201, which includes settingimage acquisition parameters. For example, image acquisition parameters,such as electron beam dwell time, electron beam landing energy, beamspot size, etc. In some embodiments, dwell time may be the biggestfactor affecting image quality and should be the same for all imagesacquired for use with the method 200. For example, a short (e.g., 0.3 μsdwell time) helps limit or reduce drift occurring during imageacquisition and further speeds up the image acquisition process.Limiting drift helps keep the underlying image signal, e.g., the imagewithout the noise, the same for images of a same location, and theincreased speed of the image acquisition process result in obtainingdata faster and may further limit sample damage, if it is a sensitivesample.

Process block 201 may be followed by process block 205, which includesacquiring training image pairs. The training image pairs may be pairs ofimages of a same spot of a sample, and each image in each image pairwill have different noise. The number of training image pairs acquiredmay be relatively small, especially in relation to the number of sampleimages acquired later, but the number acquired may also be affected by asimilarity of image noise used to pre-train the denoiser model, e.g.,ANN.

The method 200 further includes the process block 203, which includesproviding a pre-trained denoiser. The performance of this block may beperformed in parallel with, before or after process blocks 201 and 205.The pre-trained denoiser model is a pre-trained ANN, such as ANN 114,trained to denoise images. In general, the pre-training may be doneusing any noisy images and do not need to be of the same sample typeimages in process block 205 or even have the same type of noise.However, the more similar the training images are to those acquired inprocess block 205 with respect to noise, the quicker the re-training orupdating the denoiser may be.

Process block 203 may be followed by process block 207, which includesupdating the denoiser using the training image pairs. The denoiser,i.e., ANN 114, is updated using images having noise similar to the noisethat will be included in the sample images. By updating the denoiserwith images having similar noise to that of the sample images, thedenoise more robustly provides clean images, e.g., reduced noise images,in return. Further, since each training image pair is of the same spotand only differ due to noise, the denoiser learns the noise and is morecapable of removing the noise without affecting the rest of the image.

The method 200 further includes process block 209, which includesacquiring a large data set of sample images. The sample images should beacquired using the same acquisition parameters used to acquire thetraining image pairs to ensure the sample images include similar noise.The process block 209 may be performed any time after process block 201is performed, such as before, after or in parallel with process blocks203 through 207. The large data set of sample images will be of the samesample as used to obtain the training image pairs, but will cover a muchlarger area or volume of the sample. In general, the number of sampleimages will be substantially more than the number of training imagepairs. For example, the number of sample images may easily be over 10 kimages and result in terabytes of data.

Process blocks 209 and 207 may be followed by process block 211, whichincludes denoising the sample images using the updated denoiser. Thedenoising of the sample images using the denoiser, e.g., ANN, may bedone as the sample images are obtained by the ANN couple to the chargedparticle microscope, or the denoising of the images may be performed ata user's desktop computer coupled to access the sample images and theANN.

FIG. 3 is an example method 300 for sparse image reconstruction inaccordance with an embodiment of the present disclosure. The method 300may be implemented by a charged particle microscope system, such assystem 100, or by one or more computing environments coupled to receiveimages from a charged particle microscope system. In general, the method300 may reconstruct sparse images to provide a complete image. Forexample, an image including 10 to 20% pixels having information may bereconstructed to provide full 100% of the pixels with information of thesample.

Additionally, the method 300 may be implemented by an ANN, such as ANN114, using similar techniques employed to denoise images as discussed inmethod 200. The sparse reconstruction uses a similarly trained ANN tothat of the method 200 with only a slight difference between thetraining image pairs used in the methods 200 and 300.

The method 300 may begin with process block 301, which includesacquiring sparse reconstruction training image pairs of a sample. Thesparse reconstruction training image pairs will be pairs of images ofoverlapping spots of the sample, but they are not of the exact samespot. Each pair should have an overlapping area and an non-overlappingarea so that the ANN is trained to fill in the sparse or absent areasfrom images. For example, one image of a training pair may include datafrom pixels 1, 3, and 5 and the other image of the training pair mayinclude data from pixels 2, 4, and 5. Alternatively or additionally, thetraining image pairs are of the same spot on the sample, but eachpicture of a training pair uses/includes a different mix of pixels fromthe image of that spot such that there is no overlap in pixels. As inthe method 200, the number of sparse training image pairs may be small,especially compared to the number of sample images.

Process block 301 may be followed by process block 303, which includesupdating an artificial neural network using the sparse training imagepairs. The training pairs will be provided to a pre-trained ANN to beupdated for sparse reconstruction. The pre-trained ANN may be partiallytrained to reconstruct images and may be trained using sparse images andassociated full images, for example.

Process block 303 may be followed by process block 305, which includesacquiring a large data set of sparse sample images of the sample. Thesparse images may be acquired using the same parameters as used toacquire the training image pairs. While process block is shown to beperformed after process block 303, in other embodiments, process block305 may be performed prior to or in parallel with process block 303.

Process block 305 may be followed by process block 307, which includesreconstructing the sample images using the updated artificial neuralnetwork. The updated ANN will reconstruct the sparse images bydetermining what the missing data should be and included it with anassociated output image.

FIG. 4 is an example image sequence 400 showing denoising of images inaccordance with an embodiment of the present disclosure. The imagesequence 400 includes a training image pair and an associated denoisedimage provided by an updated denoiser. The images 402 and 404 areexample noisy images of a same location of a sample, e.g., a trainingimage pair, and are used in updating a denoiser, such as in processblock 207 of method 200. The image 406 is an example denoised image ofthe same location having been denoised by the updated denoiser. Whilethe example image 406 is blurry in some areas, the overall imageincludes much less noise than the noise shown in images 402 and 404.

FIG. 5 is an example image sequence 500 showing sparse reconstruction inaccordance with an embodiment of the present disclosure. The image 502is a sparse image of a sample and may be part of a large data set ofsample images and/or one image of a training image pair. In contrast,the image 504 is a reconstructed image provided by a trained ANN, suchas by process block 307 of method 300.

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), or network processing units (NPUs)that are persistently programmed to perform the techniques, or mayinclude one or more general purpose hardware processors or graphicsprocessing units (GPUs) programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, FPGAs, or NPUs with custom programmingto accomplish the techniques. The special-purpose computing devices maybe desktop computer systems, portable computer systems, handhelddevices, networking devices or any other device that incorporateshard-wired and/or program logic to implement the techniques.

For example, FIG. 6 is a block diagram that illustrates a computersystem 600 upon which an embodiment of the invention may be implemented.The computing system 600 may be an example of the computing hardwareincluded with CPM environment 102, such a controller 112, imagingplatform 108, sample preparation platform 110, and/or servers 106.Additionally, computer system 600 may be used to implement the one ormore neural networks disclosed herein, such as ANN 114. Computer system600 at least includes a bus 640 or other communication mechanism forcommunicating information, and a hardware processor 642 coupled with bus640 for processing information. Hardware processor 642 may be, forexample, a general purpose microprocessor. The computing system 600 maybe used to implement the methods and techniques disclosed herein, suchas methods 200 and 300.

Computer system 600 also includes a main memory 644, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 640for storing information and instructions to be executed by processor642. Main memory 644 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 642. Such instructions, when stored innon-transitory storage media accessible to processor 642, rendercomputer system 600 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 600 further includes a read only memory (ROM) 646 orother static storage device coupled to bus 640 for storing staticinformation and instructions for processor 642. A storage device 648,such as a magnetic disk or optical disk, is provided and coupled to bus640 for storing information and instructions.

Computer system 600 may be coupled via bus 640 to a display 650, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 652, including alphanumeric and other keys, is coupledto bus 640 for communicating information and command selections toprocessor 642. Another type of user input device is cursor control 654,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 642 and forcontrolling cursor movement on display 650. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 600 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 600 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 600 in response to processor 642 executing one or more sequencesof one or more instructions contained in main memory 644. Suchinstructions may be read into main memory 644 from another storagemedium, such as storage device 648. Execution of the sequences ofinstructions contained in main memory 644 causes processor 642 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 648.Volatile media includes dynamic memory, such as main memory 644. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge,content-addressable memory (CAM), and ternary content-addressable memory(TCAM).

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 640. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 642 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 600 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 640. Bus 640 carries the data tomain memory 644, from which processor 642 retrieves and executes theinstructions. The instructions received by main memory 644 mayoptionally be stored on storage device 648 either before or afterexecution by processor 642.

Computer system 600 also includes a communication interface 656 coupledto bus 640. Communication interface 656 provides a two-way datacommunication coupling to a network link 658 that is connected to alocal network 660. For example, communication interface 656 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 656 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 656sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 658 typically provides data communication through one ormore networks to other data devices. For example, network link 658 mayprovide a connection through local network 660 to a host computer 662 orto data equipment operated by an Internet Service Provider (ISP) 664.ISP 664 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 666. Local network 660 and Internet 666 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 658and through communication interface 656, which carry the digital data toand from computer system 600, are example forms of transmission media.

Computer system 600 can send messages and receive data, includingprogram code, through the network(s), network link 658 and communicationinterface 656. In the Internet example, a server 668 might transmit arequested code for an application program through Internet 666, ISP 664,local network 660 and communication interface 656.

The received code may be executed by processor 642 as it is received,and/or stored in storage device 648, or other non-volatile storage forlater execution.

In some examples, values, procedures, or apparatuses are referred to as“lowest”, “best”, “minimum,” or the like. It will be appreciated thatsuch descriptions are intended to indicate that a selection among manyused functional alternatives can be made, and such selections need notbe better, smaller, or otherwise preferable to other selections. Inaddition, the values selected may be obtained by numerical or otherapproximate means and may only be an approximation to the theoreticallycorrect/value.

What is claimed is:
 1. A method to denoise charged particle images, themethod comprising: Acquiring, with a charged particle beam, a pluralityof training image pairs of a sample, wherein each training image of eachof the plurality of training image pairs are images of a same locationof the sample, and wherein each image of the plurality of training imagepairs are acquired using same acquisition parameters, wherein theacquisition parameters at least includes a dwell time of the chargedparticle beam; updating an artificial neural network based on theplurality of training image pairs; and denoising a plurality of sampleimages using the updated artificial neural network, wherein theplurality of sample images are acquired using the same acquisitionparameters as used to acquire the plurality of training image pairs. 2.The method of claim 1, wherein noise in each image of the plurality oftraining image pairs is noise specific to a charged particle microscopeused to acquire the training image pairs.
 3. The method of claim 2,wherein the acquisition parameters affects the noise specific to thecharged particle microscope used to acquire the training image pairs. 4.The method of claim 1, wherein noise in each image of the plurality oftraining image pairs is similar to noise in each image of the pluralityof sample images.
 5. The method of claim 1, wherein updating anartificial neural network based on the plurality of training image pairsincludes updating the artificial neural network based on noise specificto a charged particle microscope used to acquire the plurality oftraining image pairs and the plurality of sample images.
 6. The methodof claim 1, further including: acquiring the plurality of sample images,wherein the sample images cover a larger area of the sample than doesthe plurality of training image pairs.
 7. The method of claim 1, furtherincluding; providing a pre-trained artificial neural network, thepre-trained artificial neural network not trained on images having noisesimilar to that in either the plurality of training pairs or theplurality of sample images.
 8. The method of claim 1, wherein a numberof images of the plurality of training image pairs is less than a numberof images of the plurality of sample images.
 9. The method of claim 1,wherein the plurality of training image pairs include spot images atdifferent locations on the sample.
 10. A system comprising: a chargedparticle microscope imaging platform; and a controller at least coupledto control the charged particle microscope imaging platform, thecontroller coupled to a non-transitory computer readable mediumincluding code that, when executed by the controller, causes the systemto: acquire a plurality of training image pairs of a sample, whereineach training image of each of the plurality of training image pairs areimages of a same location of a sample, and wherein each image of theplurality of training image pairs are acquired using same acquisitionparameters, wherein the acquisition parameters at least includes a dwelltime of the charged particle imaging platform; update an artificialneural network based on the plurality of training image pairs, theartificial neural network included with or coupled to the system; anddenoise a plurality of sample images using the updated artificial neuralnetwork, wherein the plurality of sample images are acquired using thesame acquisition parameters as used to acquire the plurality of trainingimage pairs.
 11. The system of claim 10, wherein noise in each image ofthe plurality of training image pairs is noise specific to a chargedparticle microscope used to acquire the training image pairs.
 12. Thesystem of claim 11, wherein the acquisition parameters affects the noisespecific to the charged particle microscope used to acquire the trainingimage pairs.
 13. The system of claim 10, wherein noise in each image ofthe plurality of training image pairs is similar to noise in each imageof the plurality of sample images.
 14. The method of claim 10, whereinupdating an artificial neural network based on the plurality of trainingimage pairs includes updating the artificial neural network based onnoise specific to a charged particle microscope used to acquire theplurality of training image pairs and the plurality of sample images.15. The system of claim 10, wherein the computer readable medium furtherincludes code that, when executed by the controller, causes the systemto acquire the plurality of sample images, wherein the sample imagescover a larger area of the sample than does the plurality of trainingimage pairs.
 16. The system of claim 10, wherein the computer readablemedium further includes code that, when executed by the controller,causes the system to provide a pre-trained artificial neural network,the pre-trained artificial neural network not trained on images havingnoise similar to that in either the plurality of training pairs or theplurality of sample images.
 17. The system of claim 10, wherein a numberof images of the plurality of training image pairs is less than a numberof images of the plurality of sample images.
 18. The system of claim 10,wherein the plurality of training image pairs include spot images atdifferent locations on the sample.